Skip to content

dev#1

Open
rsasaki0109 wants to merge 151 commits into
mainfrom
develop
Open

dev#1
rsasaki0109 wants to merge 151 commits into
mainfrom
develop

Conversation

@rsasaki0109
Copy link
Copy Markdown
Owner

No description provided.

rsasaki0109 and others added 30 commits September 5, 2025 00:00
…recalculation

Major improvements to RTK positioning implementation:

- Fix filter initialization to use RINEX header position directly
- Implement ECEF to geodetic coordinate conversion (WGS84)
- Recalculate geometric ranges in updateFilter using current filter state
- Fix ambiguity initialization to use current geometric range
- Add outlier rejection for state updates > 10km to prevent divergence
- Add convert_rtklib_to_kml.py utility for visualization

Results:
- 118 stable FLOAT solutions (no divergence)
- 0.3-0.4m horizontal accuracy compared to RTKLIB reference
- Outlier rejection successfully prevents catastrophic divergence

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
…ncy)

Major milestone: libgnss++ is now a fully self-contained modern C++17 GNSS
positioning library with no runtime dependency on RTKLIB.

RTK Performance:
- Kinematic (1.2km): 100% fix rate, 12mm RMS (exceeds RTKLIB 98.3%)
- Short static (36m): 99% fix rate, 90mm RMS
- Long static (3.3km): 52% fix rate with WL-NL AR

New self-contained implementations:
- Eigen-based Kalman filter (algorithms/kalman.hpp)
- LAMBDA integer least-squares (algorithms/lambda.hpp + lambda.cpp)
- Saastamoinen troposphere model (models/troposphere.hpp)
- Klobuchar ionosphere model (models/ionosphere.hpp)
- Coordinate conversions with geodist (core/coordinates.hpp)
- Consolidated constants (core/constants.hpp)
- Solution writer (io/solution_writer.hpp)
- Native SPP with WLS + iono + trop (spp.cpp)

RTK algorithm improvements:
- Separate rover/base satellite positions from respective pseudoranges
- Analytical Sagnac correction in geodist()
- Wide-lane/Narrow-lane AR for long baseline ionosphere mitigation
- Melbourne-Wubbena cross-validation of LAMBDA integers
- Fix-and-hold with direct state constraint (PSD-preserving)
- Position-based hold validation
- High-variance DD pair exclusion (median-relative threshold)

RINEX 3 support:
- Navigation file parsing (G/R/E/C system chars, 4-digit year)
- Observation file parsing (epoch markers, per-satellite lines)
- Broadcast ionosphere parameter extraction (ION ALPHA/BETA, GPSA/GPSB)

Infrastructure:
- Regression test suite (tests/run_regression.sh)
- Analysis tools (tools/rtk_stats.py, plot_rtk.py, compare_rtklib.py)
- CLAUDE.md development guide
- Updated README.md with actual performance data
Adds a multi-site driver around the Phase D-3 atmospheric tidal
loading single-site bench (PR #68), mirroring the pole-tide
multi-site harness (PR #69) and the sub-daily-EOP multi-site
harness (PR #75):

  apps/gnss_ppp_iers_atm_tidal_loading_multisite_bench.py

The site config schema accepts a per-site `atm_tidal_loading`
override on top of the campaign-wide `common.atm_tidal_loading`.
Real campaigns use per-site TU Wien atmospheric loading service
coefficients; for bench fixtures, sites can share a single
synthetic coefficient file via `common.atm_tidal_loading` until
real per-site values are fetched.

The driver's docstring documents the ATL coefficient file format
(matching PPPProcessor::loadAtmosphericTidalLoading) so users can
build a synthetic mid-latitude fixture for order-of-magnitude
bench checks.

TSKB + GRAZ at 2026-04-15 (DOY 105, 600-epoch cap, IGS final
SP3 + CLK from BKG mirror, common synthetic mid-latitude ATL
fixture S1 = 0.8 mm radial / S2 = 0.4 mm radial):
  TSKB: max=300µm, median=213µm, median_dz=-83µm
  GRAZ: max=347µm, median=168µm, median_dz=-89µm

Both sub-mm at mid-latitudes, both with consistent median_dz
sign — within the IERS §7.1.5 envelope.

Registers in apps/CMakeLists.txt and apps/gnss.py
(ppp-iers-atm-tidal-loading-multisite-bench). Updates
docs/iers-integration-plan.md §7 Phase D-3 with the multi-site
bench summary.
…llup) (#77)

Closes the per-effect bench cascade (#57 / #63 / #65 / #66 / #68 /
#75 / #76) which all measured per-epoch DELTA between IERS-on and
IERS-off arms. Those confirm each model is faithful, but they do
not say whether the integrated stack delivers a better ABSOLUTE
position against an external reference.

Adds:
  apps/gnss_ppp_iers_truth_bench.py
  apps/gnss_ppp_iers_truth_multisite_bench.py

Both run gnss_ppp with all IERS defaults ON, take the converged
static-mode tail of the .pos solution, and compare to the RINEX
OBS header's APPROX POSITION XYZ (which IGS stations keep at the
published ITRF coordinate). --ab also runs an all-IERS-OFF arm
and reports the residual delta. The multi-site driver mirrors
the pole-tide / sub-daily-EOP / atm-tidal multi-site harnesses
(PR #69 / #75 / #76).

TSKB + GRAZ at 2026-04-15 (DOY 105, 1500-epoch caps,
--converged-tail-epochs 600, IGS final SP3 + CLK from BKG
mirror, finals2000A.daily for EOP):
  TSKB: IERS-on 3D = 2.975 m, IERS-off = 2.978 m, delta = +2.9 mm
  GRAZ: IERS-on 3D = 2.370 m, IERS-off = 2.374 m, delta = +4.0 mm

IERS-on closer to truth at BOTH sites with mm-level improvement,
matching the per-effect bench scale — so the integrated IERS
stack is faithful end-to-end.

The meter-scale absolute residual is a SYSTEM finding: with the
current BRDC + IGS final + no IONEX / no DCB setup, PPP_FLOAT
converges to a position offset by ~2-3 m from truth, dominated
by orbit / clock / atm modeling outside the IERS scope. Closing
that gap (DCB ingestion, PPP-AR, longer convergence, mixed-
product handling) is a separate workstream.

Registers in apps/CMakeLists.txt and apps/gnss.py
(ppp-iers-truth-bench / ppp-iers-truth-multisite-bench). Adds
docs/iers-integration-plan.md §6.1 with the rollup story.
rsasaki0109 and others added 10 commits May 9, 2026 21:03
Two parsers in src/core/navigation.cpp silently rejected real
products from IGS analysis centers:

1. IONEXProducts::loadIONEXFile capped each TEC value line at 60
   characters via `line.substr(0, 60)`. Standard IONEX packs 16
   I5 values per 80-char line, so the parser dropped the last 4
   values per line. With a 73-value-per-row map (5° longitude
   step from -180 to +180), no row ever reached the expected
   value count and current_map.rows stayed empty — every TEC map
   was discarded and loadIONEXFile returned false. CODE / IGS
   final IONEX (e.g. COD0OPSFIN_*_GIM.INX from
   igs.bkg.bund.de/root_ftp/IGSac/products/) hit this path,
   making --ionex unusable in PPP. Switch to using the full line
   so multi-line value collation works.

2. DCBProducts::loadFile assumed a fixed BIAS/SOLUTION row
   layout `BIAS PRN [STATION] OBS1 OBS2 ...` with PRN at
   fields[1]. GFZ / GBM Bias-SINEX uses
   `BIAS SVN PRN [STATION] OBS1 OBS2 ...`, putting the SVN at
   fields[1] (e.g. `G080`) and the PRN at fields[2] (e.g. `G01`).
   parseSatelliteToken on `G080` failed and the entry was
   skipped, so every GFZ DCB row was dropped and loadFile
   returned false. Disambiguate by token length: SVN is 4 chars
   (`<sys><digit><digit><digit>`), PRN is 3
   (`<sys><digit><digit>`); take whichever 3-char field at
   fields[1] or fields[2] parses as a valid PRN, and shift the
   OBS1/OBS2/UNIT/VALUE/STDEV indices accordingly.

Both bugs were caught by the new IERS end-to-end truth bench
(PR #77): pointing it at a real CODE INX or GBM BSX produced
"failed to initialize PPP processor" silently. The parsers now
load successfully (25 TEC + 25 RMS maps from a real CODE final
IONEX, every GBM DCB entry).

Two new gtest cases pin the fixed behaviour:
- PPPTest.IONEXProductsLoadFullWidthDataLines exercises a
  19-value-per-row map split across a 16-value line + a 3-value
  continuation, mirroring the line-spanning path.
- PPPTest.DCBProductsLoadBiasSinexWithSvnAndStationColumns uses
  a GFZ-style ` DSB G080 G01 ... ` row.

Existing IONEX / DCB tests still pass; PPP suite 22/22 green.

NB: applying the loaded products via existing PPP code paths does
not yet improve TSKB residual against ITRF (PR #77 baseline 2.97m
→ 3.26m with --ionex). That is a separate downstream issue with
the bias / iono application logic and is out of scope here.
PreciseProducts::interpolateOrbitClock was returning false whenever the
query time matched a clock-only entry — the case for nearly every PPP
epoch when a 30-s IGS CLK file is loaded alongside a 15-min SP3 file.
The lower_bound short-circuit at line 870 ("if upper->time == time,
before=after") collapsed onto a CLK row whose position_valid was false,
so the function fell through to the single-sample branch and returned
false on the empty position. PPP's applyPreciseCorrections then silently
fell back to broadcast ephemeris for ~99% of receiver epochs.

Search the bracketing position-valid pair and the bracketing clock-valid
pair independently. Position is interpolated against the SP3 grid; clock
is interpolated against the CLK grid. New PPPTest regression covers a
query at an exact CLK-only timestamp.

Also iterate light-travel-time in PPPProcessor::applyPreciseCorrections
on the precise path so sat_position lies on the orbit at emission time
(broadcast path already iterated). The downstream geodist() applies the
Sagnac correction analytically.

Bench impact (TSKB DOY 105 / IGS final SP3+CLK, no ANTEX):
  baseline       3.23 m residual vs RINEX header
  this fix       3.82 m residual vs RINEX header
The +0.6 m drift exposes a separate downstream model bias (likely
clock-reference vs primary-signal mismatch); the broken precise path
was previously masking it via broadcast-eph fallback.
calculatePhaseWindup() existed in ppp_utils but was never called from the
PPP filter, leaving carrier-phase observables uncorrected for the
geometric rotation between satellite and receiver dipoles. The accumulating
~cm-class bias becomes substantial for sub-meter PPP and was contributing
to the residual 3-4 m offset measured at TSKB DOY 105.

Changes:
- calculatePhaseWindup now takes the sun position in ECEF, builds the
  satellite body frame from the spacecraft–Sun plane (yaw-steering
  attitude as used by IGS/RTKLIB), and uses an N–E–U receiver basis.
- The 2π ambiguity is resolved by rounding (previous − dphi) to the
  nearest integer rather than the prior `while (... > 0.5)` loop, which
  silently lost cycles when the inter-epoch step exceeded half a cycle.
- applyPreciseCorrections() computes the sun position once per epoch,
  then per satellite calls calculatePhaseWindup with the cached prior
  accumulator and subtracts the IF-combination correction
  windup_cycles × c/(f1+f2) from carrier_phase_if (or λ1 in
  per-frequency mode).

Bench: TSKB DOY 105 static PPP, IGS final SP3+CLK, RINEX header truth.
3D residual at tail: 3.82 m (PR #79 baseline) → 2.79 m (-1.03 m).

Adds two regression tests covering the 2π-ambiguity handling and the
zero-sun fallback.
The dual-frequency IF combination already cancels the first-order
ionospheric delay analytically (c1·I1 + c2·I2 = 0 by construction of
the IF coefficients). Subtracting an IONEX-derived correction on top
of an IF observable is mathematically a no-op but exposes the filter
to numerical-precision and TEC-mapping noise from the IONEX product.

Gate the IONEX subtraction in applyPreciseCorrections so it only fires
when (a) IF combination is disabled, (b) only a primary frequency is
available, or (c) per-satellite STEC estimation mode is on (which uses
IONEX as a tight state constraint, not an observation correction).

This is defensive — observed delta is small after PR for phase wind-up —
but cleans up the architecture so future iono-related model work
doesn't have to reason about a code path that should never fire.
Extend the existing receiver-only ANTEX loader so satellite blocks
(BLOCK IIA/IIR/IIF/IIIA/etc.) also load into PPPProcessor. Each entry
captures VALID FROM/UNTIL plus per-frequency PCO in the spacecraft body
frame (north→x, east→y, up→z; up = boresight toward Earth).

The matching application path is gated behind a new
ppp_config_.apply_satellite_antenna_pco flag, default OFF. Reason:
under the IGS convention switch in 2017, IGS final SP3+CLK products
are already published at the iono-free combination antenna phase
centre, so adding the body-frame PCO double-applies the offset.
Verified empirically on TSKB DOY 105 (2026-04-15): with the flag on
and the textbook (sat_pos += pco_ecef) sign, residual rises from
2.787 m → 3.207 m. A negated-sign run gives 2.633 m, but reproducing
the RTKLIB peph2pos sign — sign that DOES correct CoM-referenced SP3
to APC — empirically degrades the IGS final case, confirming the
convention finding.

Future SP3 sources known to ship CoM-referenced positions (some legacy
AC products, some GLONASS-only analyses) can opt in to the shift.

The loader itself is unconditional once --antex is supplied: it
populates satellite_antex_offsets_ for diagnostic/tooling use even when
the PCO is not applied to the geometry.
Investigation into the residual ~1.7 m gap to RTKLIB rnx2rtkp ppp-static
on TSKB DOY 105 (1.12 m vs our 2.79 m) localised the dominant
contributor: PPPProcessor::constrainStaticAnchorPosition() blends the
position state with a hard 50 % pull toward the SPP-derived anchor each
epoch and additionally zeros position×{clock, trop, ambiguity}
cross-covariances. This caps absolute static accuracy at the SPP
anchor's 2-3 m floor regardless of how clean the precise products are.

Disabling the blend on the precise-products path empirically diverges
(receiver-clock and zenith-trop states run away to 30 km within hours,
because the underlying observation model is leaking ~10–40 m systematic
into first-epoch pseudorange residuals).

This change just exposes the apply_static_anchor_blend knob (default
true, matching the legacy behaviour) and rewrites the comment so future
work has a clear place to start: identify the observation-model bias
that drives the divergence, then flip this flag false to lift the
anchor floor.

Bench unchanged at default (TSKB 3D = 2.787 m).
Two related observation-model fixes that close ~0.85 m of the gap to
RTKLIB rnx2rtkp on TSKB IGS final products (2.787 m → 1.934 m).

1. LTT (light-travel-time) extrapolation at SP3 boundaries
   PR #79 added a re-query of the precise-products interpolator at
   emission_time = reception − τ.  At the first epoch (reception time
   = first SP3 sample) the implied emission epoch falls 76 ms before
   the SP3 file starts.  The interpolator's single-sample fallback
   then returns sat_position at reception (no LTT correction) AND
   sat_velocity = 0, silently overwriting the bracket-derived
   velocity from the first call.  The geometric range was therefore
   off by sat_velocity·τ projected onto the line of sight — up to
   65 m on G01 at TSKB, which the EKF then absorbs into the trop
   state, walking trop to the 30 m clamp and forcing the receiver
   clock to swallow the rest.

   Replace the second SP3 query with a first-order Taylor expansion
   sat_position(em) ≈ sat_position(rcv) − sat_velocity·τ, which is
   accurate to mm for sub-second travel times and is well-defined at
   the SP3 boundary because sat_velocity from the first call is
   computed against the next bracket.  Per-sat std of the GPS prefit
   residual at TSKB epoch 1 collapses from 28 m to 8 m.

2. Phase-row gating threshold
   The phase rows in formMeasurementEquations were gated behind
   `convergence_min_epochs` (= 20 = 10 min at 30 s sampling) when
   precise products were loaded, vs `phase_measurement_min_lock_count`
   (= 1) on the broadcast path.  The static-anchor blend pins position
   to the SPP floor over those 20 epochs, so by the time phase rows
   activate the position covariance has been reset to 9 m² with all
   cross-covariances zeroed and the cm-class phase signal can no
   longer drive convergence.  Use the same threshold as the broadcast
   path.

Bench (TSKB DOY 105 IGS final SP3+CLK, 24 h static):
  baseline (PR #80 stack)       2.787 m
  + LTT Taylor extrapolation    2.065 m
  + phase from epoch 2          1.934 m
  RTKLIB rnx2rtkp ppp-static    1.121 m

GRAZ moves only marginally (2.20 → 2.19 m) because the LOS projection
of sat_velocity·τ depends on satellite geometry; TSKB's epoch-1 sky
view exposes the bug clearly.  Remaining 0.81 m gap to RTKLIB is
likely from the linear (vs Lagrange) SP3 interpolation away from
sample boundaries — separate work.

271 tests pass.
Replace the linear SP3 interpolator with a degree-9 Lagrange polynomial
(RTKLIB-style Neville's algorithm, 10-sample window) and disable the
static-anchor blend by default.

Linear interpolation between 15-min SP3 samples has theoretical sagitta
of 57 km at the chord midpoint (R·Δθ²/8 for GPS at Δθ = 7.5°); the LOS
projection produced km-class first-epoch pseudorange residuals (epoch 2
prefit: 4700 m on G01 at TSKB) that the static-anchor blend was
absorbing by pinning the position state at the SPP floor. With Lagrange
the same epoch-2 prefit drops to <11 m, the EKF converges cleanly with
no anchor needed, and the absolute residual approaches RTKLIB.

Bench (DOY 105 IGS final SP3+CLK, 24 h static):
                                  TSKB         GRAZ
  baseline (PR #80 stack)        2.787 m      2.200 m
  + LTT Taylor extrapolation     2.065 m      —
  + phase from epoch 2 (44c5f17) 1.934 m      2.186 m
  + Lagrange SP3 (this commit)   1.286 m      1.703 m
  RTKLIB rnx2rtkp ppp-static     1.121 m      —

The anchor flag stays in the config; it's still useful with broadcast
or SSR products whose orbit accuracy can be too poor for the filter to
converge unaided, but the default flips to false because Lagrange makes
the precise-products path RTKLIB-class without it.

Implementation notes:
- Neville's algorithm is numerically stable for the 8100 s × degree-9
  parameter range. Time origin is shifted to the query so the recursion
  evaluates at x = 0.
- Velocity is computed by re-evaluating the same polynomial at query+h
  (finite difference; degree-9 polynomial is smooth enough that h = 1 s
  matches the analytic derivative within the precision of the bench).
- The orbit and clock grids stay separate (SP3 at 15 min, CLK at 30 s);
  each interpolated by its own Lagrange window. The 30-s clock grid is
  smooth enough that the polynomial fit collapses to near-linear there.

271 tests pass.
Splits the original 1050-line ``apps/gnss_dd_residuals.py`` into one
thin CLI orchestrator plus six single-responsibility helper modules so
each layer can be unit-tested in isolation and reused by future tools
that need only a subset of the pipeline:

- ``gnss_dd_residuals_records`` — record schema helpers
  (``frequency_label``, ``pair_label``, ``residual_sigma_m``,
  ``normalized_residual``, ``abs_normalized_residual``, ``rounded``,
  ``compact_row``).  Pure functions over the dict-shaped CSV row.
- ``gnss_dd_residuals_io`` — CSV reader / worst-pair writer with a
  ``optional_float`` helper for the optional variance columns.  Pure
  I/O, no statistics.
- ``gnss_dd_residuals_filtering`` — ``filter_records(kind=, frequency_index=)``.
- ``gnss_dd_residuals_statistics`` — ``percentile``, ``group_records``,
  ``residual_stats``, ``coverage_stats``.  Numpy-free.
- ``gnss_dd_residuals_summary`` — assembles the summary dict consumed by
  the CLI and HTML renderer (``summarize_groups``, ``summarize_time_series``,
  ``compact_residual_points``, ``summarize_records``).
- ``gnss_dd_residuals_html`` — HTML / SVG renderer (``svg_line_chart``,
  ``svg_bar_chart``, ``svg_residual_scatter``, ``svg_pair_heatmap``,
  ``stats_table``, ``write_html_report``).  CSS lifted into a module-level
  constant so the renderer body stays scannable.

The CLI keeps the same ``--input_csv`` / ``--summary-json`` /
``--top-pairs-csv`` / ``--html-report`` / ``--top-n`` / ``--kind`` /
``--frequency-index`` / ``--require-*`` surface so downstream callers and
CI scripts continue to work without changes.

Tests added (six unittest modules, 69 cases total):

| Module                                          | Cases |
|-------------------------------------------------|-------|
| ``tests/test_gnss_dd_residuals_records.py``     | 14    |
| ``tests/test_gnss_dd_residuals_io.py``          | 9     |
| ``tests/test_gnss_dd_residuals_filtering.py``   | 7     |
| ``tests/test_gnss_dd_residuals_statistics.py``  | 11    |
| ``tests/test_gnss_dd_residuals_summary.py``     | 9     |
| ``tests/test_gnss_dd_residuals_html.py``        | 14    |

``python3 -m unittest discover -s tests -p test_gnss_dd_residuals_*.py``
runs the full set in <10 ms.  End-to-end smoke against a synthetic CSV
generates an identical summary dict and a 24 KB self-contained HTML
report compared with the pre-split implementation.
@rsasaki0109 rsasaki0109 added status:conflict Needs merge or rebase before it can be merged status:integration Long-running integration branch or PR labels May 23, 2026
rsasaki0109 and others added 4 commits May 24, 2026 05:56
Add the second split slice from the draft MADOCA branch: an opt-in MADOCALIB parity link path, native helper parity wrappers, guarded oracle wrappers, and CI coverage for the linked parity build.
Add the next focused MADOCA split slice: an opt-in MADOCALIB postpos bridge, gnss_ppp bridge CLI wiring, default-build guards, and a linked CI smoke using the public MIZU one-hour sample.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:conflict Needs merge or rebase before it can be merged status:integration Long-running integration branch or PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant